-
Notifications
You must be signed in to change notification settings - Fork 90
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Use exponential backoffs on secret source errors. #732
Use exponential backoffs on secret source errors. #732
Conversation
952bb8d
to
e08748b
Compare
Previously, the back off duration was based on a fixed duration + some jitter. This PR introduces exponential back offs for all secret syncing controllers. The back off will be calculated and honored whenever an error is encountered while fetching from a secret source e.g: Vault, HCPVS.
e08748b
to
6ace089
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM from my initial pass
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
👍
I wonder if it would make sense to log the backoff settings on startup? Just to make it easier for users to see what's set.
- add new metric runtime configuration gauge
Thanks! I made that change in 15e14c9. I also added a new Prometheus metric that includes the same info:
|
Previously, the back off duration was based on a fixed duration + some jitter. This PR introduces exponential back offs for all secret syncing controllers. The back off will be calculated and honored whenever an error is encountered while fetching from a secret source e.g: Vault, HCPVS. The back off configuration is controlled via some new command line arguments:
Or through the Helm chart values: